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ABSTRACT 

This paper describes six existing evaluator«*auditor 
working foraats and the conditions vhich foster credibility of 
evaluation findings. Evaluators vere classified as: (1) aeaber of 
project developmental teas, accountable to project director; (2) 
independent internal evaluator, accountable to systea in general but 
not to project directors, and (3) external evaluator, responsible by 
contractual agreement to system management* Auditors vere classified 
as either external or internal to the system* Financial and temporal 
factors, human resources, and expertise are discussed as possible 
constraints to the assurance of credible findings for each of the six 
evaluator-auditor working relationships. (Author) - 
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Alternative Methods for Assuring Credibility 
of Research and Evaluation Findings In Project Evaluation 

Always of Interest to the educational researcher or evaluator as well 
as the consuming public Is the credibility of Information provided to 
decision makers. The value of information provided to impact the decision- 
making process is probably judged first and foremost by the extent to which 
it is deemed credible. Credible information is information which accurately 
estimates the parameters under investigation and ^hich is obtained through 
a carefully and expertly designed and implemented study. An expertly 
designed and implemented study accounts for extraneous noise as much as 
possible and has built-in safeguards for assuring accurate analysis and 
leporting of the data. The purpose of this paper is to examine alterna- 
tive ways of ascuring credibility of evaluation informatiop. and to show 
the relative strengths and weaknesses of each. 

With the advent of the ESEA Title I and other federally funded pro- 
grams, educational research and evaluation has grown dramatically. This is 
particularly evident in the large urban school districts where competent 
research and evaluation departments now exist, while only a few short years 
before the research and evaluation effort in such settings was limited 
mostly to conducting testing programs. 

Largely through the efforts of Title I evaluation reports, the nation 
was made aware of the plight of the "educationally disadvantaged" r dent 
in the United States. Such stunning disclosures of the failure of edu- 
cation to effectively reach all children left many questions unanswered. 
"Why can't Johnny read?" "How did our schools get in this mess?" "What 
can be done to liiq>rove the situation?" Out of this questioning period 



came a new key word - accountability. Quickly championed by many, the 
main thrust of accountability was^ and is, meaningful disclosure of school 
performance. Such an attitude created a demand for more evaluation data 
and Information. 

It was apparent to many, that If research and evaluation Information 
was to be used, mechanisms needed to be established to assure tha validity 
of the information reported. Some educators opted for evaluators to be In- 
dependent from control of that which they were evaluating. Others argued 
that only an educational program audit similar to the fiscal audit used 
in business could assure valid evaluation information. Glass proposed 
a strong accountability model which included all aspects of accountability 
proposed up to that time. (Glass, 1972) According to Glass^ plan, school 
districts were to disclose all information relative to petformance. Then 
a comparative evaluation component would be established by which schools 
would be expected to devise alternative solutions to problems. Finally, 
such solutions would be designed along accepted research and evaluation 
guidelines with the entire efforts of the district monitored by an 
Independent auditing agency. 

Thus the public's demand for accountability in education has brought 
about the need for more valid information. It is the authors* contention 
that evaluators and auditors must do everything possible to assure valid 
information. Regardless of their best efforts the credibility of both 
evaluation and audit information is greatly effected by the locus of 
control of those doing the reporting. For example, the further removed 
the evaluator or auditor is from being controlled by those being evaluated 
the more likely the information is going to be viewed as credible. However, 
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greater credibility la gslned at the expense o£ a loss of utility, Infor- 
aatlon with high utility Is pertinent co the needs o£ the project manager 
for planning, controlling and evaluating the project. It must be reported 
in an easily manageable form and must be timely to the decision-making 
process. Locus of control can effect credibility and utility of infor- 
mation as will be shown by examining the role of the evaluator. 

Probably the best exain>le of the evaluator who is controlled by those 
being evaluated is the project evaluator who is part of the project team. 
Such a project team is usually controlled by a project manager who is often 
dependent upon project success for continued employment, or at least has 
a large degree of ego involvement. It is not too difficult to see how 
information provided by the evaluator, in spite of best efforts, is some- 
what suspect • Yet the evaluator who is part of the project team produces 
information of the highest utility to the project manager, particularly if 
the project is, in part, producing a product such as a new curriculum. In 
the design mode, the fact that the evaluator is a participating member of 
the project team permits more relevant information collection, better pro- 
ject planning, and more rapid reporting of information. Thus, when the 
evaluator is directly controlled by those being evaluated a loss of credi- 
bility of evaluation information is hopefully offset by its high utility 
to the project. 

In the large urban school systems of the United States, departments or 
branches for research and evaluation huve evolved. These research and evalu*- 
ation groups are generally not responsible for project management but for 
evaluation on^y. This removes the evaluator from control by the project 
manager. In this setting the evaluator i i responsible for dual reporting 
of information — to project management anc^ higher level administration. 



Since the evaluators continued employment is not dependent upon project 
success the evaluation information is viewed with a high degree of credl-- 
bllity. As was stated before, this high degree of credibility is obtained at 
the cost of a loss of utility. Removal of the evaluator from the project 
team will usually reduce the utility of the evaluation information. This 
most often happens because the evaluator is responsible for evaluating 
two or more projects and as a consequence commitment is divided. This 
becomes especially evident when the projects require extensive process 
evaluation. In the case of the department of research and evaluation 
supplying and controlling the evaluator, a great deal of credibility has 
been gained at the loss of some utility of information. 

In many school districts and in nearly all statewide or nationwide 
projects, a consulting firm is contracted to perform project evaluation. It 
is felt that such total independence of the evaluator lends a great deal 
of credibility to the evali:ation reports. This credibility is gained 
by a nearly total loss of utility of information, especially for the 
project manager. More will be said about the concept of external evalu- 

9 

ator in a later section of the paper. 

Figure 1 depicts what has been explained previously. As the locus of 
control of the evaluator moves from internal to the project to external 
from the organization, the utility for the project manager decreases while 
the general credibility of the evaluation information increases. This is 
assuming that the evaluator in each case is equally well trained to conduct 
research and evaluation in the public school setting. 
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The authors believe the problem of high utility-low credibility of 

information reported by the project contioller evaluator prompted interest 

in the audit function mentioned earlier. The concept seemed reasonable. A 

firm or individual would be contracted to check and verify evaluation 

designs and reports. This would give the best of desirable outcomes — 

information with high utility ond high credibility. However, the audit 

concept had one assumption very basic to making it functional. The 

assumption is best stated by Jackson and Webster. 

The success of educational program auditing in 
improving educational evaluation and reporting 
at. the local level is, to a great extent, 
dependent upon the machinery established at the 
state level to Insure competent auditors are 
available with the necessary prerequisite skills 
in the required areas of educational research 
and evaluation, management systems, and measure- 
ment. If individuals who do not possess the 
prerequisite skills are permitted to sell them- 
selves as educational auditors, the future of the 
concept is doubtful (1971, p. A). 
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Vhere Is the certification machinery to verify the capability of 
external auditors and evaluators? How effectively is it being used? At 
both the national and state level no established procedures exist for the 
certification of evaluators and in the state rf Texas no certification pro- 
cedures exist for auditors* 

A lack of verification of capabilities of external evaluators and 
auditors presents a serious problem in regards to the credibility of 
information produced by such individuals. It is completely plausible 
that a pd^orly trained or incompetent external evaltiator or auditor can 
produce highly suspect Information simply through ignorance. It takes 
but very few exposures of such instances to greatly reduce the public's 
confidence in the concept of an external evaluator or auditor, and 
possibly Af evaloation in general. 

The authors would like to present a different version of the edu^.ational 
program auditor - the internal auditor. Such a concept is often used within 
business and industry to check the veracity of reports and operations in 
general. Such an individual or group should be hired by the highest person 
responsible for research and evaluation within the organization. The 
internal auditor should be certified as to possession of the essential 
competencies and trained in any areas shown to be deficient. The internal 
auditor should be completely independent of everyone possible within the 
organization, possibly responsible only to the highest officer in the organi- 
zation. It is realized that the internal audltMSn^would be faced with many 
problems while trying^to function within an organization and yet remain inde- 
pendent and objective. There is one advantage -the internal auditor would 
quickly have over any ext«ynal auditor knowledge of the organization or system. 
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A lack ^^MofM«ei^ at least In large, urban school districts, greatl3it:«y 
reduces any effectiveness an external auditor might have, no matter how 
expert. Knowledge of the system allows data collection to be greatly 
expedited simply by knowing the various sources of Information. It also 
provides one with the necessary power to either remove or bypass 
obstacles which always seem to get in the way of data collection and 
verification. 

Figure 2 shows how combining an auditor with the evaluation function 
effects the utility and credibility of ev^^-ation information. It assumes 
the present non-existent mechanism for verification of external auditors. 



Figure 2 

Utility and Credibility of Evaluation Information 
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Because of the low utility of the information provided, the external 
evaluator (cells c and f ) is not considered a valuable asset to the organl 
zation. However, as shown in Figure 2, an internal auditor working in 
conjunction with an external evaluator could improve the utility of infor- 
mation by forcing the evaluator to examine appropriate variables and 
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monitoring the implementation of the evaluation. 

The difficulty an external auditor (cells d, e, and f) has functioning 
within an organization and the concomitant moderate credibility of infor- 
mation provided makes for a less than optimal component of the evaluation- 
audit process. This leaves the evaluator as a member of the project 
team (cell a) or a member of the organization (cell b) and the auditor 
internal to the organization as the only two acceptable loci combinations 
for auditor-evaluator. While both of these operational plans are acceptable 
the high cost of an internal evaluator (i.e., one evaluator per project) 
would, in many cases, prohibit this arrangement. Also, it is felt that, 
as a member of the project team, the internal evaluator might La pressured 
to investigate only those areas of the program which would be favorable 
to management. While the independent internal auditor would tend to 
correct this problem it is felt that an evaluator working external to 
the project but within the organization would have edeiuate freedom to 
pursue all reasonable lines of investigation. The utility of the infor- 
mation provided by this type of evaluator is rated from high to moderate 
and is a function of the work load placed on the evaluator. The information 
will be of high utility if the evaluator and support staff are assigned 
to one, rather than several, projects. 
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